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ABSTRACT 

Four research projects were condu 
improvement of individualized instruction. Two me 
foreign language were examined. In the first, the 
profile of the student's previous performance in 
and it developed a strategy to teach the student 
words. The second project tested the e^fectiven^ 
keyword association method to maximize the retenb 
Russian vocabulary. The second two projects were 
courses in- computer programing: Algebraic Interpr 
(AID) r and BASIC Jlnstructional Progxam (BIP) . In 
combines the student's history and the structure 
construct the optimal teaching strategy. (EMH) 
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the application of* a mnemonic technique called the keyvcrd method. Experinients 
witK Spanish and Russian items showed that the method could be a powerful a3[.d 
in building and retaining a large vocabulary of foreign words. \ ^ 

Two CAE courses In computer programming were developed, ffhe first pro-\ 
vldeS instruction and practice in the Algebraic Interpretive Dialogue (AED) . 
language, and was used to investigate various optimizati-on and indivldualiza-, 
^' tion techniques, on the level of single problems' 'as well as 'over entire ^ 
lessons. The BASIC Instructional Program (KEP) was developed as a vehicle fof^ 
CAE research in optimal selection of instructional material by means of an ' ^ 
internally stored cun^^Lculum description and a model •reflecting the student^s" ^ 
changing state of knowledge. Both the student history and the curriculum i 
organization are u^ed to select problems in a dynamic way, as ^ach is ^rectly 
, related to the content of the course, described as sets of specific programming 
skills. * • 

Each project is discussed briefly, with references to the. publications • 
that describe the research in detail. 
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-SUMMARY 



^This report summarizes the research conducted at the Institute for 
Mathematical Studies in th^ Social Sciences by Professor' Richard C. 
M.inson and his st^ff on ONR -Cgntract No.^ N000lii-67-A-06l2-005U, August 

V 

1970 to July 1975- The central theme of the research is' the improvement 

0 

of instruction, with reference to a theoretical framework gf optimi2;ation 
of the learning process, and specific applications in computer-assisted 
instruction (CAI). 

A the^piy of instruction is measured against the following criteria: 
(1) a model of the learning pryo'cess; (2) specification of admissible 
instructional actions; ^(3) specification of instructional objectives; 
(U) a measurement scale that permits costs to be assigned to each of the 
instructional actions and payoffs to the achievement of ' instructional 
objectives. To the extent that these four cl-lteria can be formulated 
explicitly, optimal instructional strategi?es can be derived/ Four 
projects, in tvo distinct^ subject areas, were carried out vith the 
purpose of applyiife e^^nding tne concept of a theory, of instruction. 

- Two projects iWi^vg^the^^ acquisition of foreign-language vocabulary 
items. The first (using German vocabulary) concerned itself ,,with opti-. 
raizing the computer- controlled selection of items "for study, where the 
optimization criterion was the number of items retairled on a posttest. 
The optimal strategy deyelop^ was based on a mathemati-cal model of ^ 
vocabulary learning; the model is used to compute, on a 'trial- by- trial . 
basis, an individual's current state of learning. Based on these 'comr 
putations, items were selected to optimize the^ level of learning 
achieved at the end of the instructional session. • . 



\ . ' ■ ... 

A second vocabulary acquisition project 'concerned the development 
•"**^^and application of a mnemonic ^technique called the keyword me-Qhod. The 

technique requires the student^^ to construct a chain of two links betweerf 

* ** \ 

the foreign word and ijts English translation. An audio link connects 

the foreign word to an English word with a similaf sound (the keyword); 

an imagery link connects the keyword to the English^ translation by means 

items showed that the method could be a powerful aid both in building a 
large vocabulary in a short time and in retaining the material for recall 
on delayed tests. 

Considerable research effort was devoted to two CAI courses in 
computer programming. The first course provided instruction and practice 
in the Algebraic Interpretive Dialogue (AID) language. The AID course 
was ^ap^x^tempt to apply instructional theory to a full ^c ale curriculum, 
^^^pposed to small scale experimental situations. ""Of major interest 
was the ability of the CAI program to provide instruction aDd at the 
..^ame time to record precise, extensive data on student behavior. These 
data wer&^^ti^^. zc analyze various uptimization and individualization 
techniques, on ^the level of single prcblems as well as over entire 
. iQtsscns. ' . ''^-x^ ' 

The BASIC Instructional Program (BIP) wa^ developed as a vehicle ■'^ ^ 

for CAI research in optimal selection of instructional material by means 
or an internally stored curriculum description and a model reflecting ^^.^ 
tl\^ student's ch^inging state of knowledge and skill. s^BIP's design is ^ ; 
vei^' (^iffe'rent from that of the AID course, Specifically In its decision- i' 
making mechanisms that present material on the basis of -fehe student *s ^ 
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concept-ori^/ited history rather than in a series .ordered lessons. A 

••\ • • - ♦ 

student's, history in the' AID cour-s€ consisted of a record of correct and 

incorrect responses to the problems in each lesson. In BlP^both the 

student history and the . curriculum organization are used in a much more 

dynamic way^ as each' is more directly relate/ to the content of the 

course, 'described as sets of ' very specific programming skills'.' 

Our work in these four areas is outlined and discussed briefly in 

the present report, with numerous references, to detailed discussions in 

our technical reports and other publications. 
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THE It-IPROVEMENT AND INDIVIDUALIZATION OF /iXl^UTER- ASSISTED • *• 
/ J ' INSTRUCTION: FINAL REPORT 

^ Marian Beard, Avron Barr, Dexter Fletcher, and "Richard C.' Atkinson 

The Institute for Mathematical Studies in the..^ocial-^5cie;ices was 
given research support under Office'of Naval Research Contract No. 
NQ00li+-67-A-0012-005U from August i^^^er through July 1915, to investigate 
teciiniques In^ computer-assisted instruction (CAI), particularly aimed 
tfitward the optimization and individualization of instruction. Against 
. the background of a theory of instruction, wo?k w^s conducted in .two 
subject areas: computer-controlled programs in seqpnd- language vocabulary 
acquisition and CAI i.n computed programming, Gloving comparison of the 
quite different models of learning and optimization procedures appro- 
'priate to each. ^ , ' , ' 

Atkinson (1972a, 1975a) discusses the factors that need to be 
examined in deriving optikal instructional strategies, and identifies 
the key elements of a theory of instruction. The derivatiorv of an 
optimal strategy requires thai, the instructional problem be stated in 
a form amenable to a decision-theoretic analysis. Analyses based on 
"decision ;fcheory vary somewhat from field to field, but the same formal 
ements can be found in most of them. Stated in a general way, these 

ele\ents.are as follows: , 

♦ 

The 'possible states of nature ^ • 

2. .\hfe actions that the decision- maker can take to transform the 
^i^te of nature. ' , ^ 

3. The^^^ansformation of the sta^e of nature that results from 
eaoh -e^ction. 

•^^ / . 
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^ The co^t of each o^ction. 
5« The return resulting from each state of nature. 
In the context of instruction, these elements divide naturally Unto 

■ ■ ■ L"^ 

three grcJups. Elements 1 and 3 ^^e concerned with a description 'of the 
learning process; elements ^and 5 specify the cost-benefit dimensions 

L . . ... r ■ 

of the problem"; and element 2 requires that the instructional a'ctions 

from which th^* decision maker is free to choose be precisely specified. 

♦ 

For the decision problems that arise in instruction, elements. 1 
and 3 require 'that a model of the learning process -ex4:st. It is usually 
natural to identify the states of nature with the learning stat^ss of the 
student.^ Speci^ing the transformation of the |;tates of nature caused 



by .the; actions of the decision mailer is tantamount to constructing a' 
.model of Jearning for the^ situation under consideration^ The learning 
model will"b?^fo to the >extent that ijhe state of .learning is 

imperfectly observable |)r the transformation or the sta'te of learning 
Jhat a given instr ictiojial actiqn will cause is not completely predictable^ 

, r The speci^cBtlen. (fr costs and 'returns in an instructional situation 
(elements k anc 

term basis, bu^ virtually intractable over tne long term. For the short 
term *one can assign costs and^ returns for the mastery of, say, certain 
basic reading skills, but sophisticated determinations for tfte ' long-term 
value of these Skills to the individual and Society are. difficult to 
make. There is aiTTLmportant role for (Retailed economic analyses of the 
long-term infpact of education, but such studies deal with issues at a 
more* global level than we shall consider here. The present analysis 



tends to be straightforward when examined on a shorts,-* 



•will be 'limited to those costs.. and returns directly related to a specific 
instructional ♦task* 

\ Element 2 is critical in determining the effectiveness of a decision- 
theory analysis; the nature of this element can be indicated by an.example* 
Suppose we want to design a supplementary set of exercises f or apxinitial 
reading program that involves both sight-word identification and phonics. 
- Let us assume that two exercise formats have been developed, one for 
training on sight words^ *the other for phonics. Given these formats, 
there are. many ways to design an o^v^rall program, y A variety of optimi- 
zation problems can be generated by fixing some' features *of the curriculum 

♦ and lea\ring others to be determined in a theoretically optimal manjier. 

/ ' ^ ' 

For -example, it may be desirable Ho jdetermine how the time available for 

• instruction should be divided between phonics and sight-word recognition, 
Mlth all other featares uf the curriculum fixed. A more complicated 
questiofi would be to determine tkfe optimal ordering of the two types of 
exercise^ in addition to the optimal allocation of time. It would be 

.easy to continue generatirig different optimization problems in this 
' manner. The mai:i point Is that varying the set of actions from which 

the decision maker is free to choose changes the decision. problem, even 

^ " ' / 

though the oth&r element^-- remain the same. ^ ' 

. Once these five elements have been specified, the next task^is t^ 

derive the optimal strategy for the lelrning model that best describes 

the situation. If more than one learning model seems reasonable a priori^ 

then competing candidates for the optimal strategy can be deducecj. When 

\ these /tasks have been accomplished, an experiment 'can be designed to 

.determine which strate^ is best. There are several possible directions 

r ' / ...^ . ' / 



in which^to proceed after the initial comparison of strategies, depending 
on the results of tlje experiment. 

, , CRI.TERIA FOR A THEdRY OF INSTRUCTION • ' 

■ ■ - J ' 

Our discussjLon to this point can bf sumiTiarizea by listd^ four 
criteria that must be*satisfied prior to the^ derivation of an optimal^ • 
instructional strategy: 

1. Spe'Cification of admissible instructional actions. , 

2. Specification of instructional objecfives. 

3. A measurement sc^le that permits costs to be af^gned to each 
r of the instructional actions an^ payoffs to the achievement 

of instructional obj^ectives. 

4« A model of the learning pifocess. . ... 
If these four elements can be given a precise intei^retation, then-1^ is 
generally possible to derive an optimal instn<|fcional policy. The stiiu- 
tion for an optimal policy is not guaranteed, but in recent years some 
powerful tools have been developed, for discovering optimal or near 
optimal procedures if they exist. V 

The four criteria listed above, taken in conjunction with methods 
for deriving optimal strategies, define^ either a model of instruction or 
a theory of instruction. ^Vlrei^her the term theory or model is used ' 
depends on the genel-ality of the applic^€ibns that can be made. Much- of 

•the work supported by the contract has be^concerned with the develop- 
ment of specific models for ^^pecific instructional tasks; hopefully, the 
collection of such models wil3^,provide the groundwork for a general theory 

yof instruction* ' ( . * ^ 

/ / 

In terms of the criteria listed above,"^ is clear that a model or 
theory of instruction ^ixs, in fact, a special case of what has come to be 



% known in the mathematical and engineering literature as o^fcjfflaJ^^ntrol 
theory or^ more simply, control th^/^iyT ^The development of control 
thexDry has progressed at a rapid rate both in the United States and ^ 
abroad, but most of the applications involve engineering or economic 
systems of one type or another. Precisely, the same pi'oblems are posed 

♦ 

in the area of instruction except that the sy&tem. to be controlled is 

the human learner, rather th^in a machine 9r, group of industries; To the— i 

extent thaj| the above four elements can bq formulated explicitV^ methods 

of control theory o&n bemused in deriving optimal instructional strategies. 

. , * 1 

SEWHD-LMGUAGB~-VaCABULARy ACQUISITIOH 

c ^ * ^ ^"^ ■ ^ 

Two projects involving secor^d-language vocabulary were carried out 

under the contract. " The first programs discussed ^here are. based on 

solid mathematical theories of simple learning t^asks. In particular, 

they attempt to optimize' the memorisation of translations of foreign 

language vocabulary items by xndiv.idua3,i2ing tfie sequence of item pre- 

sentation. A description of tV.i^ ins-rractional situation as a probabil- , 

istic Markov process is used to derive an item sequencing algorithm -yiat 

facilitates signifi cant improvement in acquisition rates. We' also ' , 

describe a mnem^.nxc merr^irizatior. technique that we are currenj:iy .explor- 

ing in conjuncrii\^ wi+h^tne second- language vpcabulary studies. 

An Experiment on (^timal- Sequencing Sciiemes 

In this study a large set of Germari-Enjglish item^ are to be learned^ 

during aA instructional session that in^/olves a sei^ies ^of trials. On 

each trial, one of the German words is presented and the student, attei^ts 

to give the English translation; the correct translatfon^ij^ then pre 

sented for a brief stui^y period. " A ^?jl:e determine d numbe/T of trials. is 

' \ ^ ' ^ 1,3 




allocated for the instructional session, and afteT^apme intervening 

c / 

period a test is administered o^r the" entire vQcabulai^. The problem 
is to specify a strategy for presenting items during the instructional 
session so that perfoiroance on the delayed test' vill be maximized. 

Four strai;egies f^^sequencing the instructional^ material vill be 
^considered. One strategy, designated RO for randoij} order^ is'^to cycle 
through the set of items randomly; this strategy is not expected to ^"ke % 
particularly effective, but it provi(|es a benchmark against which to . 
evaluate fottier procedures. A seccSid strategy, designated SS for self 
selection, is to let 'the student determine for himself hov best to . 
sequence the material. In this mode, the student decides on eacl^strial 
whj^ch item is to be presented. 

iPhe third ^and fourth schemes 'are based on -a decision-theoretic 
-analysis- of the task. ' A matftematicai model that provides an accurate^ 
account of vocaliulary acquisition is assumed to hold in the^ present 



situation. The modeiris used to compute, on a trial-by-trial basis, an 
individual student 's "current state, of -learning. Based on these compu- 
tations, items are s^n^ed for test and stT^^^o as, to optimize the^ 
level of learning achieved at the termination of the in5i:«tCf5sonal . t 
session. Two optimization strategies derived from this type of analysi; 
will be examined. In one'ca^ the computations for determining an 
'optimal strategy are carried out assximing that all vocabulary items are 
of equal' difficulty; this strategy is designated OE (i'.e., optimal unde 
thST a^umption of equal item difficulty). In the othei' case, the. compu 



.tation^ take into account variations in difficulty level among items; 
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this strategy is called OU (i.e., optimal under the assumption of unequal 
item difficulty). The details of. these two strategies will be described 
later. " A ' 

3oth the OU and CE (Schemes assume that vocabulary learning can be 
described by a fairly simple model. We postulate that a given item is 
in one*§fJ:hree states (P, T, an^ U) at any moment in time. *If the item 
is in State P, ttien it;D translation is known and this knowledge is ^ 
"relatively" permanent in the sense tli'at the 'learning of other items ^ 
will" not interfere with it. If the item is m State. T, then it is also 
known but on a "temporary" basis; in State "R^the learning of* other. items 
can give ;rise to interference e^fect^ that cause the itgm to -be for- . \ 

gotten. In State U the item is not ^known,'-"SnartHe' student is unable to 

■ . ^ • " ' . "v. , ■ 

give a translation. 

fc , • ■ • 

When Item i i^ DreBentea the following^ transition matrix describes : 

• % 

the possible change in .its state: „ • ■ t ^ 

P T U ■ 

P' 1 - 0 0 ^ 

L'i) - r x(i.) 1-xOi) V 0 

U y(:) l-y(i)-2(i) 

, " TH- 
ROWS of the ma\r-l>r repr^-ient tne *5*^ate of the item at the start of the 
» 

trial/ and .column^^ the st^te a.t the end of the trial. Or^ trial when 
*some other Item is preoe.ite-i for test and study,^ transitiorisj in the 
state of Item i also may t^^ piace. Such, transitions can occu>-ohly 
if the student makes an error ori the other item; in that case^ the tran- 
sition matrix applied to Item i is as follows: 

' ' . • ^ 

, A • 
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P li 0 .0 

F{i) = T 0. X-f(l) ' . f(t) 
U 0 Q 1 



Basically, the idea is that wherivSome other item is presented that the 
student does not know, forgetting may occur for Ttf>m i i-r i t. is in State T 



' ^ Prior to conducting the expejlmerrt Iceported here, a pilot study^^v^ 
run using the word lists and, the RO procedui^ descHbed above. Data 

f rorai the pilot study were employ^.^ to estimate the * parameters of the • 
model; the estimates were obtained using the minimum chi-square procedures 
described, in Atkinson (l972b). ^JTwo separate estimates of parameters were 
made. In one Qase it was assumed that the items were all equally diffi- 
cult^ and dfiita from all 8^+ iten\s were lumped together to obtain a single 
estijnate of ^ the. parameter vector; this estimation procedure will be 



called ^theJ^qual-parameter case (E case). In the second case data wer^ 
separated by items, and an estimate of the parameter vector was* made for 
each of the 8^ items; this procedure will be called the unequal-parameter 
case (U case). The two seta of paramet-er estimates were then used to 
generate the optimzation schemes* previously referred to as the OE and 
Ol>»procedures. * 

. In order to formulate an "optimal" instructional strategy, it is 
necessary to be precise abbut the quantity to be maximized. For the\ 
present experiment the goal is to maximize the total number of items | ~ 
•the student' correctly translates on the delayed test. To do this,'Ve 
need to specify the relationship between the state of learning at the 
end of the instructional session *and perfoxmance on the delayed test. 
The aissumption made here is that only those items in State P at the end 



of thgr; instructional session will be , translated correctly on the delayed 
- test; an item in State T is presumed to be forgotten d'urd^rig the inter- 
vening week. Thus, the problem of maximizing delayed-t^'st performance, 

involves maximizing the number of items in State P tat the end of the 

/ * * 

instructional session. - . I ' * 

The learning model can be .used to derive equations and, in turn, 

* compute the probabilities of being in States P, a^d U fQi^^^ea^ item 
at ^he st^rt of any tri^l, conditionalized on the s/tudent*s respftri^ 
history up to th it trial'. ' Given numerical estimailes of these proba- 
bilities, a straiie^y for optimizing performgince As to select that item 
for presentation that has the greatest probability of moving into State 

f P. -This strategy has- been temed the "one-stage" optimization procedure 
because it looks alipad one trial in making decisions. 

The experiment Was carried out under computer control. The -'students 
participated in two iessiuns: an ^"instructional session" of approximately 
two hours and a bilefer ''delayed- test s^ession" administered one week 
l^atef . Tr*e delayea test was the same for all students and involved a 

^"^I^Sit over the entire" vocabulary. The', instructional session ^/.as'more 
complicated; Ti^e vocabulary items were divided into sewn lists, each 
containing' 12,Gtraan Wv.ra^; the lists were arranged in a round-robin 
order. ' oh each trial of the instiiictional session a list was displayed^ 
"and the^ student Inspected* it for a brief period of timl".-' Then one of 
the items on the libt was selected for test and study. In the RO, OE, ' 
and OU conditions the item was selected by the computer; in the SS 
condition the 'item was chose^^ b^' Ih^ student. After an item was selected 
^or^test, the student actempted. to provide a translation; then feedback 




/ 

I 

/ 




J regarding^the correct xranslatioiTvas giver^/. The next trial began 

^ the comput'er displaying the next lis^ in t^e round robin, and the same 

procedure was i^epeated. The instructional session continued in thii^ 
fashion for 336 trials. 

I ' 

The results of the experiment can ^be summarized as follows 

raance during the instructional session' is best for the RO condition, next 

best for the OE condition which is slightly better t^n the SS con*dition, 

and poorest for the OU condition. The or^er of the groups is reversed 

on th'e delayed t&a^. The OU condition is best with a correct response 

probability of .79; the SS condition is next with .58; the OE condition 

* > 

follows closely a;t .5^;* and. the RO condition is poorest at '.38. * The 

observed pattern of results is what one would exp^ect. In the SS condition 

, the students are trying to test themselves on .items they do not know; 

consequently^ during the instructional session they should have a lower' 

proportion of correct responses than students run .on the 1^0 procedure, 

. where items are tested at randt«<^^^Similarly ♦ the OE and OU ^^^nditions 

. involve a procedurie that a^teir.pts to i^j^ntify e^id test those Items that 

V. have n*6t yet beea jx^astered arid should p^r'-pdufee high error tates during 

the instructional session/ " The i^rdering of groups on the delayed test 

Is reversed oinoe all words are tested in* a non-selective fashion; under 

these conditions the proportion of corrfect responses provides a measure 

of a student's mastery of the total^set of vocabulary items. 

The magnitude of the effects observed on the delayed test is of " 

practical significance. The SS condi|.tion (when compared to the RO ■ 

condition) leads to a relative gain of 53^» whereas the OU condition 

, yields a relative gain of 10&f>*^ It* is interesting that students were 
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effective* in detetmning an optim/l study sequence^ but not so effective 
as the best of the two adaptive/teaching systems. 

The- OU procedure is sens/tive to interitem differences and conse- 
quently generates a more ei^ective optimization strategy than the OJl 
procedure. The OE proce(idre^ however, is almost as effective as having 
the student make his own instructional decisions and far superior to a 
random presentation sphere. 

*This investigayiori and similar studies are reported in detail in 
Atkinson (1972b) ^ Atkinson arid Paulson (1972), and; Paulson (1973)^, 
Mnemonic Methods/ and Vocabulary, Learning 

When conducting vocabulary studies of the sort reported above, one ^ 
s struck by /che large variability in ^learning rates across^ subjects. 
Even Stanford students^ who represent\a highly selected sample from the 
college population, display impressiveiy large between-subject differences 
These di^erences majr^/^flect differences in fund^ental learning abil- 
ities^ yut they are also influenced by the strategies that each student 

brings /to bear on the task. Jood learners can introspect with ease about 

/ ' * 

a "bag of tricks^ tr.ey'^use in vocabulary ieaming/ whereas poor learners 
/ • ' ■ 1 

are unable to describe what 1^'h^y are doing except possibly to comment 
tha^ they rehearse to themselves* The poor learner^ mi'ght well* perfonri 
aV a much higher level if they were aware of the techniques that good 
learners report using* With this in mind, we' conducted a series of 
experiments on .mnemonic methods /for vocabulary learning. In this summary 
' report we will only describe trie rtature of one of these procedures that 
we have called the keywprd method ; for a more detailed account of this 
rese^h see Raugh 'and Atkinson (1975) and Atkinson (1975b). 
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The keyword /method divides -the study of, a vocabulary item into two.^ 
stages. The f^st stage involves associating the spoken foreign word to 
an English word that sounds approximately like some part of the foreign • 
word. As an example, the Spatii'sh woifd caballo (pronounced somewhat^ like 
"cQb-eye-yo") contains a sound that resembles the spoken English word 
"eye"; we call stich a, similar sounding English word a keyword . The ' 
s.econd stage involves mental imagery in which a symbolic image of the 
keyword interacts in a graphic way with a symbolic image of the English 
translation. In'thfe.'case of eaballo (meaning horse), one could form a 
mental image of something like a Cyclopean eye winking in the forehead 
of a horse or a horse kicking a giant eyei As another example, the 
Spani*sh word -for duck is pato^ (pronounced somewhat like "pOt-o"). 
Employing the English word "pot*' as the keyword one could imagine^ a 
duck hiding under an overturned pot with its webbed feet and tufted 
l^lil sticking out below. The method, can 'be thought of as a &hain 'of 
two links connecting a foreign word to ii^s English translation through 
the mediati& cf a keyword. The foreign word is" linked to the keyword 
by a similarity in suund (tre a coustic link ) ; the keyword is in turn 
linked to -fihe English transla,tion by a 1:^ a me r- gene rated mental image 
(the mnemonic ox imagery link)* 

The experiments evaluating the effect of the keyword method with 
Spanish vocabulary Items are. reported in Raugh and Atkinson (1975). We. 
have also completed a series of similar ^studies using a Russian vocabulary 
(Atkincon & Raugh, 1975).« Iri one such experiment, two treatments were, 
compared: subjects in the "keyword group were supplied* with an English 
keyword to facilitate their learning, -while subjects in^ the control 



group were giwn no mnemonic aid.^ For all subjects, the Russian item 
was pronounced (through.^ compute r-coi:i*|:rolled audio facility) as the 
keyvord and tran^latiqn^ or the>' translation alone, were diSE|jayed on a 
CRT terminal screen. I ^ ^ 

' V 

Instructi9n sessions, lasting approximately k5 minutes per day fgr 

K 

three days^o P^sented a 120-word Russian vocabulary, one UO-wgrd sub- 
vocabulary on each day. A daily session consisted of three study-test 
cycles through the hO words. During the study phase, each Russian word 
• was pronjounced as the appropriate material was jdisplayed'^( either keyword 
and translation or translation alone) for 10 seconds. -In^ the test phas^ 
the Russian word was pronounced and the program waited 10 seconds for ^ 
the subject to initiate his* typing of ifhe English translation, .y. 

A comprehensive test of all 120 words wafe given^'on the fourth day, 
Qand'a delayed comprehensive test was givfen 30 to 60 ^ays latei^ The 
results of all the tests, durin^the 'three: instruction days and f OT- bojh 
comprehensive tests, favored the keyword co^idition. On each day the 
keyvord group learned more words in two study-test trials than the 
control group learn^^ in all three ^trials. On the comprehensive test 
(Day 1+T, the mean probability of a correct ,2*sponse was .72 for the 



keyword condition, .li6l fir the control", finally, on. the delayed c^|npre- 
herisive test, the probability correct for the keyword group wds .li3, 
for the control group, .28. ..^ * ^ * 

It appears I'^t^th^ size of yjiji.e "keyvord effect for Russiari is^ e^^ti' 

% \ * X ' "1 

.larger than that fo-r "Spanish. The Wason is that many subjects! in th$ 
Spanish experijnentb had studied at lea^t one Romance language and con- , 




sequejntly-were able to learn of vf^ Spanish words by using ^cognates 



as- memory aids. In l^us&ian there are few cognatefe, and the keyword . 
method appears to be even mojre useful. During the "last year a large 
computer-based vocabulary drill supplement using the keyword method was 
offered to second-year l^ussian slMdents at Stanford (see Raugh, Schupbach 
& Atkinson, 1975)* 



INSTRUCTION IN* COMPUTER PROGRAMMING 

This section describes two CAI\co}irses in computer * programming, 

A 



both intended for college or Junior c 

7. 



course^ which teaches xlc^e AID language, 




level students. The fi^st ■ 

iginally de.veloped with 

fy^s from the National Aeronautics and Sp^^ Administration; this 

development is described by Friend and Atkins^ (l97lV and Friend (197I) . 

Contiaued development under ONR support used th^AID course as a j:^search 

^vehicle for studies 4?. optimisation procedures appr^riate in complex 

technical areas. Development cf th^ second course, called the BASIC 

Instructional Program, has been ^upj^orted Joi^ly by ONR^ and the Advanced 

ReseabsljJProjer.ts Agency; 

The AID Cour3>> ^ 

The course ''Com>i:^er-ar*?i5:ted Xn^uAjoticn in Prog^mmirig: AID" is 

compLrtely fc^elf-contair'^/ cina ret^uir^s no s^ervisi4)n frdsji a qualified 

instructor of prograjf&aing. -^brief student manual ( Friend \l972V is 

supplied to su^gflement the ins>t:raction given by computer. Te\etyp^ 

• ^ - \ > \ 
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writer ope;?^ion is simple and csn bo learned fjrom snort instra\t\onX^ 
prin;^^d^n the student' manual. 

Once the student hasfthe telt^l^ewriter in;/6pc ration, ^IX^ further 
instruction is given by computer under- tho ..conl rol of a program known 



fi\ try 



\ 



as IHST. This prggram, 'which is the major component of the INSaJRUCT ' 
system, interprets coded" lessons^ providing individualized, tutorial 
instruction to th^ student ♦ ,,Tl?Ls instructional system and the .method 
of programming lessons for it are describe(^^ Friend (197I). 

The AID course^ uses most the feaftires of the INSTEUGT. -system. 
The course contains 50 lessons organized into/ seven lesson^ blocks. " 
Each block contains five tutorial lessons, vfollowed by "a self -test and . 
a general review. The 50th lesson i^ a conceding lesson independent 
of the blocks. The lessons' vary iri length fronKlO to 60 exercises de- 
pending upon the content. Lessons of average length require about one 
hour to complete. Lesson length is completely under student control, 
and* a student may take a few. exercises or several lessens in one sitting. 

One of the primary teaching stra-^egies* used in the course is the • 

'provifTlon fcr Gtude^t c-uitrol of the sequence of iifetruction. Students, 

may skip from any exercise thfi'^oource to any other exercise at any 

time, retracing thoir •st<jps\f tb-y wish;, or skipping lessorife eritirely. 

This btrcit'='gy is ir*t^\'.ded to onc^urag-^ lue L^tuder.t to take responsi/bility 

} * / 

for learni-g t^-- coi^copts,, not biniply I'cr progrerc.l;)£ tnrough a g/ven 

bet of e>^•^rciseL- . Most college i tuaentn are capable, ana desir^ts, of 
- ass'juning >hic. rcrsporisiLili*^y, and the provi^oion* of student control of 
in^ruction is -assumed to, provide* motivation. 

iae of thic allowance for student control, the 50 lessons may 
be taken in any ^e^uence. If the student aoih not exercise -his pre- 
rogative for choosing the sequence, ttije- lesst.rn are aut,ijm4rically*^ ^ • 
sequenced fc^r hi^; ahd it.'i/o assumed tl.at ^wxrA '-hiier^t^ will compl^te^ 
the lesions in the order indicated.' 



18 



Besides the matn strand of lessons, the course also contains review 
lessons, one for'e^ch of the tutorial lessons in the seven lesson blocks. 
These review lessons are also tutorial and cover the same concepts as do 
the lessons they are ass^iated' with. Howev6ir, they present each concept 
from a slightly 'differcnt viewjtoint, providing additional praqtice in the 
skills to be learned. In general,; each lesson covers five or six related 
concepts. In review lessons, the student may review whichever conce'pts 
he wishes, tn any order he chooses. In fact) he must choose the oi^er; 
there is no automatic sequencing provided by the program. At the end 
of eatg^ tutorial lesson, the student is asked if he wants tc? review any 
of the idea^r^overed in the lesson just completed. The student need pot 
wait for these reminders, of course, since he can call for any review, 
or any exercise in any review, whenever he wishes. ' * . 

Also' associatea witn each tutorial lesson is a summary ^of the lesson 
and the student is reminded at the end of each lesson that s\immaries are 
available at his option. In addition to the main stran^of lessons, 
the reviews, and the summaries," tuejj^Jtjg a strand of ^^ext^^ci^dlV* ^ ' 
problems* containing more difficult programming problems to be solved by 
the more capable students* ' 

The inclusion of review- lessons is a gross method for providing 
indi3dbdtlalized remediation. A moife sensitive means of individualizing 
remediation is used within the lessons themselves, where non-optional 
remedial sequences, of exercises are "given* automatically to sfudents who 
demonstrate an inadequate understanding of the material Vbeing taught. 
Because 'of/ this automatic remediation, different students may receive 

different numbers of exercises in' a given lesson. 

I ■ 



A student who makes an incorrect ^response to an exercise may not 
need an entire s^ence of remedial ^'x^rcises* He may plrofit from a 
single specific (Corrective message, pointing out the error and allowing 
him another try 'at the same problem'. Tbiis kind of specific correction 
is used for most exercises in the course. Messages are provided, not 
for, all possible incorrect responses., but for those incorrect responses 
Judged to be most Ilk 

The curriculum and student' con-^f-ol featuij^s aare described more 
fully biHB^end, Fletcher, and Atkinson (l97a>^nd Frien4^ ( 1973a, b) . 
Extensive Analysis^ of students problem-solving b^havior^ focussing on 
problem dif/iculty and diversity of student solutions to j)rogramming 
problems,^ is presented in Friend (1975)* 

An important aspect of the research in individualization* involved 

mechanisms'"'aa:i;Owing studente to exercise a considerable degree of con- 

^^^^^ " ^ ^ 
trol over the coriten^^-oiid sequencing of instractional material, as 

mentioned above. A study descrtfe^by Beard, Lorton; Se^arle, and Atkinson 

^1^3) was conducted to compare a student-sel?ctte»..acl^ against <two 

strategies of computer control. One major finding of this study was . 



.tnat'studentb- ao not :.ho':<5e \6 exercise much control over the material 
presented to thcrn,- btuiienls who were- allowed their choice of lessons 

consistently followed, the path 9!^ the ordered lessons.^ More significant, 
in light of theydirection taken since that time, was the conclusion that 

V' 

^ the AID cour-^ itself was not ideally suited to investigation of sophis- 

.* « 

ticated' individualization Schemes. First, since the curriculum is clearly^ 
laid out in a pedagogically sound; linear order, it 'actually^ discourages 
students from making diffea;«nt choices. Second, the instructional program ^ 



r 



/ 



/ 1 



is not directly linked to the AID interpreter, through which the students 
i^ritejth^ir own programs, and thus the course cannot provide assistance 



or instruction- during Ihe problem-solving activity itself. As the 

; ■" X ^ • ■ " ' \ r, ('•; 

student writes his program, his only sources of assistance are the errcirf 
messages provided by ^h^ non-instructional AID interpreter. 

An inadequacy of the AID course, especially for research .purposes,; 
is i^ limited ability to characterize individual students' knowledge / 
•of specific 'skills, and its inability to relate students' skills to the 

'\ ^ ' V 

curi^culum as anything more than a ratio q^f p.robifems correct to problems 
attempted. The program cannot n^ake fine distinctions between a student's 
stare'ngths and weaknesses, and .cannot present instructional iriatei^al 
specifically appropriate to that student beyond "harder" or "easier" 
lessons^ In orde?* to explore the effects of different curriculum selec- 
tion strategies in more detail, we developed a new introductory course 
in computer programming capable of representing both its subject matter 
and student performance more adequately. . ^ " - 

The BASIC Instriactional Program is a stand-alone, fully self- 
contained course in BASIC programming at the high "school/college level 
developed over the past two years with the assistance\of over 300 under- 
graduatg^^^o^ave taken the ccSurse at DeAnza College, the .University of 
/San Francisco, and Stanford^ Our classroom experiences developing EEP 

w 

^^s an instructionial vehiG^liB^^re ^escribed by Barr, Beard, Lorton, an4 

Atkinson (I97i^a,b). BIP's major, features are: 

- A monitored BASIC interpreter, written by the project staff, 
which allows the instructional System maximal knowledge 
. about student errors^ 
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- -A HINT system that -gives bot^-^raphic ^nd textual aid i3> problem J 

solving. ^ ' . ^' 

- Individuaiizedytask selection ^^.sed on a Curriculum Informatipn ' 

Network,, whfch descHhes thfe/proble,ms in tenns of fundamental 
skii;Ls." Problems are sel^atfed using a model of the student's 
acquisition of the skill? ,Teq,ti red- by his earlier p'rograihming 
•■/probiejhs . ' ' ' ■ f. ; 

- A curriculum consisting of approximately 100 we 11- writ ten, ; 

inteTesting^programming problenj^ at widely varying levels 
of aiffic^y. ■ • 



•, The tutorial programming laboratory environment supported by^EEP ^' . 
is- described fully by Barr, Beard, and Atkinson (1575a)-. ' To- the student 
seated at a terminal, -BiP looks much like a 'typical "timesharing 

BASIC operating system. The BASIC interpreter, .yritten especially for 
BIP, analyzes each program line after the' student ;ty^s it, and notifies 
the student of syntax errors. When the student runs '.his program, it Is 

checked fot structural illegalities, an<a-4hen, during runtime, '^execution 

- - * . ' ■ , - ■ V ' 

'errors are indicated. A file storage .system, a calculator, and utility 

commands are available. . ,.. ; . 



BIP's Cur riculum Information Network - ""■'kS^- 
— ^ -TT— , • 

In much- of; the^ GU^rrent research in tutorial CAI, generativ&'Ciil; 

- ■ , ■ • ■ . • 

and mixed-initiative natural language dialogues, fthe centra3Jj*oblem is 
the "representation" of -th^^ eubject domain, which is a^^a fundamental 
concern of ^research in cognitive' psychology and a^plcial intelligence? 
The- goal is to provide *a representation of the^Cject matter that is 
sufficient for individualized tutoring ai^'^lso has a realistic and , ^ 
nvaqageable computer implementation. I^echnical subjects,,' development ^ 
of skills requires the integration ..i#4aGts, not just their memorization, 
and, the organization of lXi^\:r\^<iy)^BX material is crucial^ for effective 

Instruction in these areas. / ^ ^ 

/ 
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^ The CuiTiculi4|ilifoiiiation Network (CIN) is inteiided to provide the 

wit 



instruct ipnal prbpr^^ with ah»exp]j^t 

of an author-wfttten curriculvim y^lBarr^, Beard, & Atkinson, ^975c-) . It 




entatior%)f the s'tructure 



contains the interrelations between the "problems wkich the- «Mf hor vould 

have used -implicitly in determining his '^branching" schemes, rts^cgjj-ows 

meaningjful modelling of the student *s progress along the ^lines of his 

developing sl?*41s, not just his liistory of right and wrong responses, 

without sacrificing the motivational advan-^ages of hum^ organization of 

the' curriculum material* F'or e^cample, in the HEP course, the* CGEN con- 

' .sists. of a complete description of e'aqh of tl^e 100 pr-ugramming problems 

in teitns of the skills developed in solving the problems. ThuSj, "the 

' . ' * ; - 

instructional program can monitor the student's progress on these skJ;^ 

'. and phoose the ne:tt task with an appropriate group of"^ n^5W-*skills. An 
^^^^ , ^ / ' ' \ ■ / * • 

intermediate step is introduced between recording the student^s history"* ^ 

ttnd^ tj^iecting his next problem: the network becomes, a model of the 

student *s. ^tate of knowledge, sin^e it. has an estimate of his ability 

<4in the relev^t skills, not just his performance on the problems he^^as ^ 

completed, branching decisions are based on this model instead of being 

^ dete3:mined simply by the student *s success/failure history on the problems 

,he has conpleted. f - ^ * ^ 

In this way-^ a problem can be presented for different purposes to ^ 

students with different histori^.* The flexi%lity of the curriculum 

is of course multiplied as a result.* ^ More importantly, the individual 

problems in the curriculum can be"^more natural and meaningful; they do 

not necessarily involve only one gkiir or technique. In frame-type 

curriculums this^one-dimensionality of the problems has a constricting 
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eWeit. ^In essence, the network as implemented in EEP is a method of 

descri]4ng.a/'real" curriculum in terns of the specific skills that can^ 

*' : i ' ' -u • 

be Identified as a student is" problem areas. 

SCP's Instructional En\dronment ' ( 

Computer programming, like most other technical subjects, is better 

.J|amed through. experience than thorough direct instruction,' egpecially 

if that expert pnce can be paced at' a speed suiteB to the indj^dual 

■ student. Throughout the EEP course, the primary emphasis is placed on 

the solution of problems presented in 'the tasks. EEP does "ot present 

( ^ » 
a sequence of instructional statements followed by questions. Instead, 

a problem is described and the student is expected to write his own 

. BASIC program to sol^ it. ^ As he develops his BASIC program for each 

tq3k, the student is directed to appropriate sections "of the student 

manual (Beard & Barr, 197^) for full explanations of BASIC state^ 

p¥5gramming 'structures, etc. He is ^so encouraged to. use 

student- orJpntpd feaiures available, such as an Interact! veVdebuggil 

• facility and various "help" options described in Barr, Bear(|,, and 

Atkinson (1975b>. 

When a student enters ifche course he. finds himself in task "GREENFI 
which requires a two-liiie program solution. The problem, as he is told, 
is worked out in great detail. in the EEP student manual. Thus, the 
trauma of being, told to "write a prograii that..."' in his first session _ 
is alleviated by following the model dialogue, in which many typica^ ■ 
mistakes are illustrjated, yet his hands-on programmW experience begins 
immediately. When he has finished the task by success^&iOj^-^ipning his 
program-, the etudent prc.ceeds by. requesting "MOEE", His progr6ss is 



srous 




i 



evaluated after- each task. In the "Post T.ask Interview" he is asked to 
indicate whether or not he needs more \^rk on the skills required by the 
task, which are"lis)ted separately for 

Ac tcon as the student completes GBEEHPLAG, therefore, the instruc- 
tional program knows something about his own estimation o'f his abilities. 
In addition, for all future tasks his solution is evaluoAed ,(by means of 
comparing its output with that of th^model solution run on the same test 
data) and th&^ results are stored with" each skill >equir|d by the task. 
The program uhen has two measutgspf the sjudent/s prcgress in each skiU- 
his' self-e^'BluafeLon and its~ow»-comparison-test results . 
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A studeny Pi*ogresses through the curriculum by^/riting, and running, 
a J)rogram tl^t solves the problem presented on his teijninal. Virtually 

are imposed on t^i^amount of time he spends, the number 
of lines ■hSywriH:es, the number of errors he is allowed to make^, the 
number of Limes he to execute the program, the changes he makes 

within it, etc. The task on whidh he is working is stored on a stack- 
like structure^ so' that he may work on another task, for whatever reason, 
and return to the previous task automatically. The curriculum structure 
can accommodate a wide variety of student aptitudes and skillsK Mqst of 
the curriculum- related options are designed with the less con^etent 
student in mind. A more independent student may simply ignorfe the options. 
Thus, HIP gives students the opportunity to determine their own "challenge 
levels" by making assistance available hut not inevitable. 

HEP offers the student consideriable flexibility in making his own 

O 

task-related decisions. He may ask for hints and subtasks to hel^ him 
get ^tarted in solving the given problem, or he may ponder the problem 

25 . - . - 



on his own^ using only th^^anual for additional information. He may 

request a different task 3By^ name/ in the event that he wishes to work on 

•^^ . ' 

it Immediately, either compl&'fing the new task or not, as he.chposes. 

On his return, EEP tey^ him the name of the again-current task., and 

allows" him to have its text printed to remind him of the problem he is 

lh solve. The student may request the model solution for any task at 

any time, but HEP wilT^not print the model for the current task unless 

the student 'has 'exhausted the, available hints and subtasks. Taken. to- 

gether, the curriculum options allow for a wide range of student prefer- , 

ences and behaviors. a • \ 

The HEP program has been running successfully with both junior 

college and university students. However, the program is still veiy 

much in ah experimental stage. From a psychological viewpoint, the 

principal research issues deal with (l) procedures ^f or obtaining on-line 

estimates of student abilities as represented in the infonnat ion, network, 

.and (2) alternative methods ^for using the current estimates in the in- 

formation nexwork to make instmctional decisions. For a more complete 

descriptioft of our recent work on BIF and a .review of our 'plans for 

continued research see Barr, Beard;^ ^nd Atkinson (l975d). . ^ 

CONCLUDING REMARKS ^ 
The projects described in this paper have one theme in common, 
namely^ developing computer-controlled procedures figiuoptimizing the 
instructional prpcess. For several of the ^inst^ctioTiar'tB^k^-eonsi^red 
here, mathematical models of the learning Jh:ocess were forraulattfJ which 
Haae it possible to xise formal methods in 4eri^ri.ng optimal policies • 
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In -other cases the "optimal schemes" were* not o^><;m&i in a well-defined 
sense, but were base^ on our intuitions about learning .and some relevant 
expetiments* ^ In a sense, the diversity represented in these examples 
o^rresponds to the state' of the art,3n the field of instructional design^ 



^For-^^^^ tasks we can use psychological theory tt>*help define optimal 
procedures; for others our intuitions, modified by e^xperiments, must 
, guide the efforts Hopefully, our understanding -of these matters will 
increase, as mere projects are^ndercaken to develop sophisticated in- 
stru^iiional programs • ^ ■ ^ 

'Some have argu^ that any attempt to. devise optimal st.rategies is 
doomed to failure, and that the learog^r him^^lf Is the best j\&ge of ^ 
appropriate instructional actions ♦ ^ We are not sympathetic to a learner- 
controlled approach to instruction, beeifee we believe its advocates are** 
trying to avoid the difficult but challenging task of developing a viable 
^-Wieory of instruction^ Thera oVviously i^ a place for the learner^s 
judgm^^ts in making. iric-tjhictioT.jal decisions^ for evainpL^., such judgirents 
play an i^p^)r^ant Jrole 'in Several par+j of c-ux FTP Ci^^rse. F/.vever, 
using.the .le^i^ie-r ' ^ jUiigrrt;r<t a^^ i ix- of. clever"- 1 item; IritV raati^.r^ in 
making instructional decisions Is .iiffer^ n^ from prupo?ing that the 
learner 'sh^calii have complete contrul- Results prei^ented i.:. Ihi-: ?>aper 
and those cited in BearJ et al. (1973) indicate that the leanier iv. not 
a particularly effective decision maker in guiding the learning process. 

At thf beginning of this !*ieporl we defined the four criteria ^h-tt 
must be satisfied before an optimal ^instructional pr^ce.rire -can be ^ie.- 
• rived us'ing formal methods. For -^he type;; of in-.^tract irrial ^^itua** ^jiV^ • 
'd^alt witn Juring tK^aife <ji this cc<nt^rao+ speJiifica+l^.rr c^;, v^ffer^ i 



for the first three elements • ' However, the fOarth element— the speci- 
fication of a model of ^he learning process— represents 'a^ major obstacle. 
Our theoretical understanding of learning is so limited that only in very 
special cases can a model be specified in enough detail to enable the 
derivation of optimal procedures ^ • Ifetil we have a much deeper under- 
standing of the learning (process, the identification of truly effective , 
strategies will not be possible. '< 

However, an all-inclusive -theory of learning is -not a prerequisite 
for the development of optimal procedures. What is needed is a model 

t^«t captures the essentiai features of that part of the learning process 
« ' • * 

being tapped by a given instructional task. Even models that have been 

rejected on tl^ bas^s of laboratory investigations may. be useful in 
deriving instructional ^strategies. Several of the learning models con- 
sidered in this, paper have proven unsatisfactory vhen tested in the 
laboratory and evalO,ated using standard goodness-of-fit criteria; never- 
theless, the optim^ strategies they generate are -often quite etfectlve. 
Our own preference! is to formulate, as complete a leamiilg model as 
intuition and data will permit' 6nd then use that model to inves#gate 
optimal procedures. When possible the learning model should be repre- : 
■ sented in- the form of mathematical equations, but otherwise as a set o^ 
statements In a computer-simulation program. The main point is that the 
development of a theory of instruction cannot progress if one holds thj' 
•■ view that a comprehensive theory of learning is a prerequisite. Rather, 
advances in learning theory will affect the development of a theory of ' 
' instruction, and conversely "the development of a thepi^ t>f Instruc^i^on 
• will influence the direction of research on learning. , 

■ . . • 28 ; . 
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